Camera stabilisation is an important technology used to improve the quality of still images and video images captured by a device. The stabilisation may be used for a stand-alone camera, or for cameras within a network configured for a wide range of applications including surveillance, event broadcast, cinematography, medical imaging or other analysis. For example, a camera network may be used in a computer vision system used to generate free viewpoint video (FVV) of objects and activity in a field of view surrounded and imaged by a network of cameras. Such an FVV system may be capable of processing video images in real time and generating virtual video footage of the scene suitable for broadcast with a low latency. Virtual video images may be generated from a variety of viewpoints and orientations that do not correspond to any of the cameras in the network.
Many camera stabilisation methods have been developed, and the appropriate method depends on the environment and requirements of a given camera system. Mechanical stabilisation systems include specialised mounts, shoulder braces and gimbals. The mechanical stabilisation systems are popular for producing a wide range of content such as cinema and live broadcast of events. The mechanical stabilisation systems are generally suitable for moving cameras, such as mounted cameras that pan and zoom to track the events on a sports field, or roving cameras that can be deployed rapidly for on the spot coverage. Mechanical stabilisation systems damp out high frequency instability. However, mechanical systems would not be expected to handle lower frequency motion such as drift.
Other image stabilisation methods that are included within a camera have also been developed. For example, optical image stabilisation is common on modern cameras and operates by varying the optical path to a sensor. Such methods are particularly effective at removing camera shake from captured images and, in particular, the kind of camera shake associated with a hand held camera. In addition to stabilisation of video sequences, optical image stabilisation and the like can improve the sharpness of individual frames by removing blur. Optical image stabilisation can also improve the performance of auto-focus which may be reduced by instability in an image capture sequence. However optical image stabilisation generally does not handle camera roll and may not be suitable for low frequency instability such as drift in a mounted, fixed camera.
Another internal camera stabilisation method varies the position of a sensor rather than the optical path. Methods which vary the position of a sensor have the advantage over optical stabilisation of being capable of correcting for camera roll. Gyroscope data or a DSP may be used to calculate the required sensor shift based on captured images. The maximum correction available depends on the maximum motion of the sensor which may limit the extent of stabilisation to large displacements such as low frequency drift in a fixed mounted camera system.
In addition to mechanical and optical stabilisation methods, a number of digital image stabilisation methods exist that rely purely on digital processing of captured images. Digital image stabilisation methods transform the image at each frame to compensate for motion of the camera. However, digital image stabilisation methods have limitations with processing the very high data rate from a modern camera which may consist of high definition images at a high frame rate. The limitations may be avoided by post-processing of video sequences. However, post-processing of video sequences may not be possible in all scenarios, such as in the case of live broadcast events where latency is critical.
The stabilisation methods described above are unable to handle high and low frequency instabilities in image capture at high frame rates and high resolution using limited hardware. Hence, there is a need for a low latency digital image stabilisation method that can handle a broad range of camera instabilities using relatively low storage and computation cost.